Filter array and light detection system

ABSTRACT

A filter array used in a light detection device, which generates image data corresponding to each of N wavelength bands (N being an integer greater than or equal to 2) included in a specific wavelength range, includes optical filters. The optical filters include various types of optical filters with different light transmittance with respect to each wavelength band. M i ≥0.1 with respect to each wavelength band, where M i =μ i −σ i , μ i  denoting an average value of light transmittance of the optical filters with respect to light having a wavelength included in an i-th wavelength band (i being an integer greater than or equal to 1 and less than or equal to N) of the N wavelength bands, σ i  denoting a standard deviation of the light transmittance of the optical filters with respect to the light having the wavelength included in the i-th wavelength band.

BACKGROUND 1. Technical Field

The present disclosure relates to filter arrays and light detectionsystems.

2. Description of the Related Art

By utilizing spectral information about a large number of bands, such asseveral tens of bands, each being a narrow band, detailedcharacteristics of a target object can be ascertained, which is notpossible with a conventional RGB image. A camera that acquires suchmulti-wavelength information is called a “hyperspectral camera”.Hyperspectral cameras are used in various fields, such as in foodinspection, biological examination, drug development, and mineralcomponent analysis.

U.S. Pat. No. 9,599,511 discloses an example of a hyperspectral imagingdevice that utilizes compressed sensing. This imaging device includes anencoder as an array of optical filters with different wavelengthdependency with respect to light transmittance, an image sensor thatdetects light transmitted through the encoder, and a signal processingcircuit. The encoder is disposed on an optical path that connects asubject and the image sensor. For each pixel, the image sensorsimultaneously detects light on which components of wavelength bands aresuperimposed, so as to acquire a single wavelength-multiplexed image.The signal processing circuit utilizes information about the spatialdistribution of spectral transmittance of the encoder so as to applycompressed sensing to the acquired wavelength-multiplexed image, therebygenerating image data for each wavelength band. In the imaging devicedisclosed in U.S. Pat. No. 9,599,511, an optical filter array having atleast two transmittance peaks (i.e., maximum values) within a targetwavelength range is used as the encoder.

U.S. Pat. No. 9,466,628 discloses an example of a filter array includinga Fabry-Perot resonator in which a dielectric multilayer film is used asa reflective layer. Japanese Unexamined Patent Application PublicationNo. 2018-107731 discloses an example of a filter array including varioustypes of color filters.

SUMMARY

One non-limiting and exemplary embodiment provides a light detectionsystem that can improve the spectral resolution of a hyperspectralcamera, and a filter array used in the light detection system.

In one general aspect, the techniques disclosed here feature a filterarray used in a light detection device that generates image datacorresponding to each of N wavelength bands (N being an integer greaterthan or equal to 2) included in a specific wavelength range. The filterarray includes optical filters. The optical filters include varioustypes of optical filters with different light transmittance with respectto each of the N wavelength bands. M_(i)≥0.1 with respect to each of theN wavelength bands, where M_(i)=μ_(i)−σ_(i), μ_(i) denoting an averagevalue of light transmittance of the optical filters with respect tolight having a wavelength included in an i-th wavelength band (i beingan integer greater than or equal to 1 and less than or equal to N) ofthe N wavelength bands, σ_(i) denoting a standard deviation of the lighttransmittance of the optical filters with respect to the light havingthe wavelength included in the i-th wavelength band.

According to the technique of the present disclosure, the spectralresolution of a hyperspectral camera can be improved.

General or specific aspects of the present disclosure may be implementedas a system, a device, a method, an integrated circuit, a computerprogram, or a storage medium, such as a computer-readable storage disk,or may be implemented as a freely-chosen combination of a system, adevice, a method, an integrated circuit, a computer program, and astorage medium. The computer-readable storage medium may include anonvolatile storage medium, such as a CD-ROM (compact disc-read onlymemory). The device may be constituted of one or more devices. If thedevice is constituted of two or more devices, the two or more devicesmay be disposed within a single apparatus, or may be disposed separatelywithin two or more separate apparatuses. In this description and theclaims, the term “device” may refer not only to a single device but alsoto a system formed of devices.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A schematically illustrates a light detection system according toan exemplary embodiment;

FIG. 1B illustrates a configuration example of the light detectionsystem in which a filter array is disposed away from an image sensor;

FIG. 1C illustrates a configuration example of the light detectionsystem in which the filter array is disposed away from the image sensor;

FIG. 1D illustrates a configuration example of the light detectionsystem in which the filter array is disposed away from the image sensor;

FIG. 2A schematically illustrates an example of a filter array accordingto an exemplary embodiment;

FIG. 2B illustrates an example of a spatial distribution of lighttransmittance in each of wavelength bands included in a targetwavelength range;

FIG. 2C illustrates an example of a transmission spectrum of a certainfilter included in the filter array illustrated in FIG. 2A;

FIG. 2D illustrates an example of a transmission spectrum of anotherfilter included in the filter array illustrated in FIG. 2A;

FIG. 3A is a diagram for explaining an example of the relationshipbetween the target wavelength range and wavelength bands includedtherein;

FIG. 3B is a diagram for explaining another example of the relationshipbetween the target wavelength range and the wavelength bands includedtherein;

FIG. 4A is a diagram for explaining the characteristic of a transmissionspectrum of a certain filter in the filter array;

FIG. 4B illustrates a result obtained by averaging the transmissionspectrum illustrated in FIG. 4A for each wavelength band;

FIG. 5A schematically illustrates a spatial pattern of lighttransmittance with respect to each of the wavelength bands in the filterarray;

FIG. 5B schematically illustrates an example of a transmittancehistogram that the filter array has with respect to an i-th wavelengthband;

FIG. 6A schematically illustrates an example of a captured image havingrandom noise superimposed on an original image;

FIG. 6B schematically illustrates the relationship of Expression (1) inwhich captured image data containing the random noise is expressed as aproduct of a matrix and spectral image data;

FIG. 7A illustrates an example of a transmission spectrum of a certainfilter included in a filter array different from that in thisembodiment;

FIG. 7B illustrates a mean square error (MSE) between a ground truthimage and a spectral image in each of five wavelength bands in a casewhere an image captured using the filter array different from that inthis embodiment contains random noise;

FIG. 8 illustrates the relationship between an average MSE between aground truth image and a spectral image in each of the wavelength bandsand M_(i)=μ_(i)−σ_(i) in a case where σ_(i)=0.08 with respect to eachwavelength band;

FIG. 9A illustrates the relationship between an average MSE between aground truth image and a spectral image in each of the wavelength bandsand M_(i)=μ_(i)−σ_(i) in a case where σ_(i)=0.05 with respect to eachwavelength band;

FIG. 9B illustrates the relationship between an average MSE between aground truth image and a spectral image in each of the wavelength bandsand M_(i)=μ_(i)−σ_(i) in a case where σ_(i)=0.07 with respect to eachwavelength band;

FIG. 9C illustrates the relationship between an average MSE between aground truth image and a spectral image in each of the wavelength bandsand M_(i)=μ_(i)−σ_(i) in a case where σ_(i)=0.1 with respect to eachwavelength band;

FIG. 10A is a cross-sectional view schematically illustrating a firstexample of the light detection device;

FIG. 10B is a cross-sectional view schematically illustrating a secondexample of the light detection device;

FIG. 11A schematically illustrates an example of a filter illustrated inFIG. 10A;

FIG. 11B illustrates an example of transmission spectra that asingle-sided distributed Bragg reflector (DBR) structure and adouble-sided DBR structure have when light vertically enters thestructures;

FIG. 12A is a cross-sectional view schematically illustrating a thirdexample of the light detection device;

FIG. 12B is a cross-sectional view schematically illustrating a fourthexample of the light detection device;

FIG. 13A is a cross-sectional view schematically illustrating a fifthexample of the light detection device;

FIG. 13B is a perspective view schematically illustrating an example ofa sub filter array and a light detection element; and

FIG. 14 illustrates an example of transmission spectra of eight types ofsub filter arrays.

DETAILED DESCRIPTIONS

Exemplary embodiments of the present disclosure will be described below.The embodiments to be described below indicate general or specificexamples. Numerical values, shapes, components, positions and connectionmethods of the components, steps, and the sequence of the steps areexamples and are not intended to limit the scope of the presentdisclosure. Of the components in the following embodiments, a componentnot defined in the independent claim indicating the most generic conceptis described as a freely-chosen component. Furthermore, the drawings arenot necessarily exact illustrations. In each drawing, same referencesigns are given to substantially identical components, and redundantdescriptions may sometimes be omitted or simplified.

Before describing the embodiments of the present disclosure, theunderlying knowledge forming the basis of the present disclosure will bedescribed.

U.S. Pat. No. 9,599,511 discloses an imaging device capable ofgenerating high-resolution multi-wavelength images, that is, images withrespect to individual wavelength bands. In this imaging device, an imageof light from a target object is encoded by an optical element called an“encoder” and is captured. The encoder has, for example,two-dimensionally-arranged regions. A transmission spectrum of each ofat least two of the regions has a maximum transmittance value in each ofwavelength ranges within an imaging-target wavelength range. Forexample, the regions may respectively be disposed in correspondence withpixels of an image sensor. In an imaging process using theaforementioned encoder, data of each pixel contains information aboutthe wavelength ranges. In other words, image data acquired in theimaging process is data containing compressed wavelength information.Therefore, the amount of data can be minimized by simply retainingtwo-dimensional data. For example, even in a case where a storage mediumis limited in capacity, long-duration moving-image data can be acquired.The multi-wavelength images are generated by performing a reconstructionprocess involving reconstructing the image captured in the imagingprocess to generate images respectively corresponding to the wavelengthranges. In the following description, each of the images generated andcorresponding one-to-one to the wavelength ranges, that is, wavelengthbands, may also be referred to as “spectral image”.

The encoder may be realized by, for example, a filter array includingtwo-dimensionally-arranged filters. For example, each filter may havethe structure of a so-called Fabry-Perot resonator that includes tworeflective layers and an interference layer located therebetween. As aFabry-Perot resonator, for example, the structure disclosed in U.S. Pat.No. 9,466,628 may be employed. The filters may be designed such that thetransmission spectrum of each filter has peaks in the imaging-targetwavelength range. Alternatively, the filters may include, for example,the various types of color filters disclosed in Japanese UnexaminedPatent Application Publication No. 2018-107731.

An image actually captured using an encoder may contain random noise,such as optical shot noise. Random noise deteriorates the spectralresolution of a spectral image generated in a reconstruction process.There is room for improvement in filter arrays that reduce the effect ofrandom noise.

A filter array according to an embodiment of the present disclosureincludes filters in which the light transmittance within animaging-target wavelength range is appropriately designed. Such a filterarray reduces the possibility in which the spectral resolution of ahyperspectral camera decreases due to random noise. A filter arrayaccording to an embodiment of the present disclosure and a lightdetection system according to the present disclosure equipped with thefilter array will be briefly described below.

First Item

A filter array according to a first item is used in a light detectiondevice that generates image data corresponding to each of N wavelengthbands (N being an integer greater than or equal to 2) included in aspecific wavelength range. The filter array includes optical filters.The optical filters include various types of optical filters withdifferent light transmittance with respect to each of the N wavelengthbands. M_(i)≥0.1 with respect to each of the N wavelength bands, whereM_(i)=μ_(i)−σ_(i), μ_(i) denoting an average value of lighttransmittance of the optical filters with respect to light having awavelength included in an i-th wavelength band (i being an integergreater than or equal to 1 and less than or equal to N) of the Nwavelength bands, σ_(i) denoting a standard deviation of the lighttransmittance of the optical filters with respect to the light havingthe wavelength included in the i-th wavelength band. N may be an integergreater than or equal to 4.

In this filter array, an effect that random noise has on a spectralimage is suppressed, so that the spectral resolution of a hyperspectralcamera can be improved.

Second Item

In the filter array according to the first item, σ_(i)≥0.05 may besatisfied with respect to each of the N wavelength bands.

In this filter array, the filters can be readily distinguished from eachother, so that the spectral-image reconstruction accuracy can beimproved.

Third Item

In the filter array according to the first or second item, μ_(i)≥0.2 maybe satisfied with respect to each of the N wavelength bands.

In this filter array, the transmittance is increased so that noiseimmunity can be improved, that is, an image reconstruction erroroccurring when noise is superimposed on image data acquired byimage-capturing can be suppressed.

Fourth Item

In the filter array according to any one of the first to third items,M_(max)/M_(min)·≤3 may be satisfied, where M_(max) and M_(min) denote amaximum M_(i) and a minimum M_(i), respectively, with respect to the Nwavelength bands.

In this filter array, a variation in the spectral-image reconstructionaccuracy between the wavelength bands can be suppressed.

Fifth Item

In the filter array according to any one of the first to fourth items,at least one optical filter of the optical filters may include aninterference layer having a first surface and a second surface oppositethe first surface, and a reflective layer provided on the first surface.A transmission spectrum of the at least one optical filter may havemaximum values in the specific wavelength range. The reflective layerneed not be provided on the second surface.

In this filter array, M_(i)≥0.1 can be satisfied in accordance with aspecific structure.

Sixth Item

In the filter array according to the fifth item, the reflective layermay include at least one selected from the group consisting of adistributed Bragg reflector and a metallic film.

In this filter array, a reflective layer that efficiently reflects lightwithin the specific wavelength range can be realized.

Seventh Item

In the filter array according to the sixth item, the distributed Braggreflector may include at least one set of a first refractive-index layerand a second refractive-index layer. A refractive index of the firstrefractive-index layer may be higher than a refractive index of thesecond refractive-index layer.

In this filter array, the reflectance of the distributed Bragg reflectorcan be appropriately designed by adjusting the number of sets of firstrefractive-index layers and second refractive-index layers.

Eighth Item

In the filter array according to the seventh item, the firstrefractive-index layer may have a thickness of λ(4n_(H)), the secondrefractive-index layer may have a thickness of λ(4n_(L)), and theinterference layer may have a thickness larger than λ/(2n_(H)), where λdenotes a wavelength included in the specific wavelength range, n_(H)denotes the refractive index of the first refractive-index layer, andn_(L) denotes the refractive index of the second refractive-index layer.

In this filter array, light having the wavelength λ can be efficientlyreflected.

Ninth Item

In the filter array according to the sixth item, the metallic film mayhave a thickness larger than or equal to 1 nm and smaller than or equalto 100 nm.

In this filter array, the transmittance of the metallic film can beincreased by appropriately adjusting the thickness of the metallic film.

Tenth Item

In the filter array according to any one of the first to fourth items,the light detection device may include an image sensor. The filter arraymay be disposed such that light transmitted through the optical filtersenters the image sensor. At least one of the optical filters may be asub filter array including two-dimensionally-arranged sub filters. Thesub filter array may include various types of first sub filters havingdifferent transmission wavelength ranges and one or more second subfilters or openings each transmitting N light beams having a wavelengthincluded in a corresponding wavelength band of the N wavelength bands. Apercentage of an area of the one or more second sub filters or openingsoccupying a total area of the sub filter array may be higher than orequal to 4%.

In this filter array, M_(i)≥0.1 can be satisfied in accordance with aspecific structure.

Eleventh Item

In the filter array according to the tenth item, the various types offirst sub filters may include a red filter, a green filter, and a bluefilter.

In this filter array, M_(i)≥0.1 can be satisfied in accordance with thefirst sub filters including RGB color filters.

Twelfth Item

A light detection system according to a twelfth item includes the filterarray according to any one of the first to eleventh items and an imagesensor that is disposed at a position where the image sensor receiveslight transmitted through the optical filters and that has sensitivityto light having a wavelength included in the specific wavelength range.

In this light detection system, a hyperspectral camera with improvedspectral resolution and increased detection light intensity can berealized.

Thirteenth Item

The light detection system according to the twelfth item may furtherinclude a processing circuit that generates the image data based on dataindicating a spatial distribution of the light transmittance of theoptical filters and compressed image data acquired by the image sensor.

In the present disclosure, each circuit, unit, device, member, orsection or each functional block in each block diagram may entirely orpartially be implemented by, for example, one or more electroniccircuits containing a semiconductor device, semiconductor IC (integratedcircuit), or LSI (large scale integration). The LSI or the IC may beintegrated in a single chip or may be configured by combining chips. Forexample, the functional blocks excluding storage elements may beintegrated in a single chip. Although the terms “LSI” and “IC” are usedhere, the terms used may change depending on the degree of integration,such that so-called “system LSI”, “VLSI” (very large scale integration),or “ULSI” (ultra large scale integration) may be used. A fieldprogrammable gate array (FPGA) to be programmed after the LSI ismanufactured, or a reconfigurable logic device that can reconfigure theconnection relationship inside the LSI or can set up the circuitsections inside the LSI can also be used for the same purpose.

Furthermore, the function or operation of each circuit, unit, device,member, or section may entirely or partially be implemented by softwareprocessing. In this case, the software is stored in a non-transitorystorage medium, such as one or more ROM (read-only memory) units, anoptical disk, or a hard disk drive. When the software is executed by aprocessor, a function specified by the software is implemented by theprocessor and a peripheral device. A system or a device may include oneor more non-transitory storage media storing the software, a processor,and a required hardware device, such as an interface.

Embodiments Light Detection System

FIG. 1A schematically illustrates a light detection system 400 accordingto an exemplary embodiment of the present disclosure. The lightdetection system 400 includes an optical unit 40, a filter array 10, animage sensor 60, and a processing circuit 200. The filter array 10 has afunction similar to that of the “encoder” disclosed in U.S. Pat. No.9,599,511. Therefore, the filter array 10 may also be referred to as an“encoder”. The optical unit 40 and the filter array 10 are disposed onan optical path of light incident from a target object 70. In theexample illustrated in FIG. 1A, the filter array 10 is disposed betweenthe optical unit 40 and the image sensor 60.

In FIG. 1A, an apple is illustrated as an example of the target object70. The target object 70 is not limited to an apple and may be afreely-chosen object. Based on image data generated by the image sensor60, the processing circuit 200 generates image data with respect to eachof wavelength bands included in a specific wavelength range (alsoreferred to as “target wavelength range” hereinafter). This image datawill be referred to as “spectral image data” in this description. Thenumber of wavelength bands included in the target wavelength range willbe defined as N (N being an integer greater than or equal to 4). In thefollowing description, the spectral image data to be generated withrespect to the wavelength bands will be referred to as spectral images220W₁, 220W₂, . . . , and 220W_(N), and these spectral images willcollectively be referred to as spectral images 220. In this description,a signal indicating an image, that is, a group of signals indicatingpixel values of pixels constituting an image, may sometimes be simplyreferred to as “image”.

The filter array 10 includes translucent filters arranged in rows andcolumns. The filter array 10 is an optical element in which the lighttransmission spectrum, that is, wavelength dependency with respect tolight transmittance, varies from filter to filter. The filter array 10modulates the intensity of incident light for each wavelength band andallows the incident light to pass through.

In the example illustrated in FIG. 1A, the filter array 10 is disposedclose to or directly on the image sensor 60. The expression “close to”implies that the filter array 10 is close to the image sensor 60 to anextent that an image of light from the optical unit 40 is formed on asurface of the filter array 10 in a state where the image is clear to acertain extent. The expression “directly on” implies that the two areclose to each other with hardly any gap therebetween. The filter array10 and the image sensor 60 may be integrated with each other. In thisdescription, a device that includes the filter array 10 and the imagesensor 60 will be referred to as “light detection device 300”.

The optical unit 40 includes at least one lens. Although illustrated asa single lens in FIG. 1A, the optical unit 40 may be constituted of acombination of lenses. The optical unit 40 forms an image on an imagingsurface of the image sensor 60 via the filter array 10.

The filter array 10 may be disposed away from the image sensor 60. FIGS.1B to 1D each illustrate a configuration example of the light detectionsystem 400 in which the filter array 10 is disposed away from the imagesensor 60. In the example in FIG. 1B, the filter array 10 is disposed ata position located between the optical unit 40 and the image sensor 60and away from the image sensor 60. In the example in FIG. 1C, the filterarray 10 is disposed between the target object 70 and the optical unit40. In the example in FIG. 1D, the light detection system 400 includestwo optical units 40A and 40B, and the filter array 10 is disposedtherebetween. As in these examples, an optical unit including at leastone lens may be disposed between the filter array 10 and the imagesensor 60. The filter array 10, the optical unit 40, and the imagesensor 60 may have atmospheric air filling the spaces therebetween, ormay be sealed with gas, such as nitrogen gas.

The image sensor 60 includes two-dimensionally-arranged light detectionelements. The image sensor 60 may be, for example, a CCD (charge-coupleddevice) sensor, a CMOS (complementary metal oxide semiconductor) sensor,or an infrared array sensor. The light detection elements may include,for example, photodiodes. The image sensor 60 may be, for example, amonochrome-type sensor or a color-type sensor. The target wavelengthrange may be set arbitrarily. The target wavelength range is not limitedto a visible wavelength range, and may be an ultraviolet, near-infrared,mid-infrared, far-infrared, or microwave wavelength range.

In the example illustrated in FIG. 1A, each light detection element isdisposed facing one of the filters. Each light detection element hassensitivity to light in the imaging-target wavelength range. In detail,each light detection element has substantial sensitivity required fordetecting light in the imaging-target wavelength range. For example, theexternal quantum efficiency of each light detection element in theaforementioned wavelength range may be higher than or equal to 1%. Theexternal quantum efficiency of each light detection element may behigher than or equal to 10%. The external quantum efficiency of eachlight detection element may be higher than or equal to 20%. In thefollowing description, each light detection element may also be referredto as “pixel”.

The processing circuit 200 may be, for example, an integrated circuitthat includes a processor and a storage medium, such as a memory. Basedon an image 120 acquired by the image sensor 60, the processing circuit200 generates data of the spectral images 220 individually containinginformation about the wavelength bands. The spectral images 220, as wellas a method for processing an image signal in the processing circuit200, will be described in detail later. The processing circuit 200 maybe incorporated in the light detection device 300, or may be a componentof a signal processing device electrically connected to the lightdetection device 300 in a wired or wireless manner.

Filter Array

The filter array 10 according to this embodiment will be describedbelow. The filter array 10 is disposed on the optical path of lightincident from a target object and modulates the intensity of theincident light for each wavelength before outputting the light. Thisprocess performed by a filter array, that is, an encoder, is referred toas “encoding” in this description.

FIG. 2A schematically illustrates an example of the filter array 10. Thefilter array 10 includes two-dimensionally-arranged filters. Each filterhas an individually-set transmission spectrum. The transmission spectrumis expressed as a function T(λ), where λ denotes the wavelength ofincident light. The transmission spectrum T(λ) may have a value greaterthan or equal to 0 and less than or equal to 1.

In the example illustrated in FIG. 2A, the filter array 10 has 48rectangular filters arranged in a 6 row by 8 column matrix. This ismerely an example, and a larger number of filters may be set in anactual application. For example, the number may be about the same as thenumber of pixels in the image sensor 60. The number of filters includedin the filter array 10 is set in accordance with the intended usagewithin a range of, for example, several tens to several thousands offilters.

FIG. 2B illustrates an example of a spatial distribution of lighttransmittance for each of wavelength bands W1, W2, . . . , and Wiincluded in the target wavelength range. In the example illustrated inFIG. 2B, the differences in the gradation levels of the filters indicatedifferences in transmittance. A paler filter has higher transmittance,whereas a darker filter has lower transmittance. As illustrated in FIG.2B, the spatial distribution of light transmittance varies fromwavelength band to wavelength band.

FIGS. 2C and 2D illustrate examples of transmission spectra of a filterA1 and a filter A2 included in the filters of the filter array 10 inFIG. 2A. The transmission spectrum of the filter A1 and the transmissionspectrum of the filter A2 are different from each other. Accordingly,the transmission spectrum of the filter array 10 varies from filter tofilter. However, not all the filters need to have different transmissionspectra. In the filter array 10, at least two of the filters havetransmission spectra different from each other. In other words, thefilter array 10 includes two or more filters with different transmissionspectra. In one example, the number of transmission spectrum patterns ofthe filters included in the filter array 10 may be equal to or greaterthan the number i of wavelength bands included in the target wavelengthrange. The filter array 10 may be designed such that at least half ofthe filters have different transmission spectra.

FIGS. 3A and 3B are diagrams for explaining the relationship between atarget wavelength range W and the wavelength bands W1, W2, . . . , andWi included therein. The target wavelength range W may be set to any ofvarious ranges in accordance with the intended usage. For example, thetarget wavelength range W may be a visible-light wavelength range fromapproximately 400 nm to approximately 700 nm, a near-infrared wavelengthrange from approximately 700 nm to approximately 2500 nm, or anear-ultraviolet wavelength range from approximately 10 nm toapproximately 400 nm. Alternatively, the target wavelength range W maybe a radio-wave range, such as a mid-infrared, far-infrared,terahertz-wave, or millimeter-wave range. Accordingly, the wavelengthrange to be used is not limited to a visible-light range. In thisdescription, nonvisible light, such as a near-ultraviolet ray, anear-infrared ray, and a radio wave, in addition to visible light willbe referred to as “light” for the sake of convenience.

In the example illustrated in FIG. 3A, the target wavelength range W isequally divided by i into a wavelength band W1, a wavelength band W2, .. . , and a wavelength band Wi, where i denotes a freely-chosen integergreater than or equal to 4. However, the example is not limited to this.The wavelength bands included in the target wavelength range W may beset arbitrarily. For example, the bandwidths may be nonuniform among thewavelength bands. There may be a gap between neighboring wavelengthbands. In the example illustrated in FIG. 3B, the bandwidth varies fromwavelength band to wavelength band, and a gap exists between twoneighboring wavelength bands. Accordingly, the wavelength bands may bedifferent from each other, and may be set arbitrarily. The divisionnumber i for the wavelengths may be less than or equal to 3.

FIG. 4A is a diagram for explaining the characteristic of a transmissionspectrum of a certain filter in the filter array 10. In the exampleillustrated in FIG. 4A, the transmission spectrum has a maximum value P1to a maximum value P5 and minimum values with respect to wavelengthswithin the target wavelength range W. In the example illustrated in FIG.4A, the light transmittance within the target wavelength range W isnormalized such that the maximum value thereof is 1 and the minimumvalue thereof is 0. In the example illustrated in FIG. 4A, thetransmission spectrum has maximum values in wavelength bands, such asthe wavelength band W2 and a wavelength band Wi−1. Accordingly, in thisembodiment, the transmission spectrum of each filter has maximum valuesin at least two wavelength bands from the wavelength band W1 to thewavelength band Wi. It is apparent from FIG. 4A that the maximum valueP1, the maximum value P3, the maximum value P4, and the maximum value P5are greater than or equal to 0.5.

Accordingly, the light transmittance of each filter varies fromwavelength to wavelength. Therefore, the filter array 10 transmits alarge amount of incident light in certain wavelength bands and does nottransmit much of the incident light in other wavelength bands. Forexample, the transmittance with respect to light in k wavelength bandsamong i wavelength bands may be higher than 0.5, whereas thetransmittance with respect to light in the remaining (i−k) wavelengthbands may be lower than 0.5. In this case, k denotes an integersatisfying the relationship 2≤k<i. Supposing that the incident light iswhite light uniformly containing all wavelength components of visiblelight, the filter array 10 modulates the incident light into lighthaving discrete intensity peaks with respect to the wavelengths for eachfilter, superimposes the multi-wavelength light, and outputs the light.

FIG. 4B illustrates an example of a result obtained by averaging thetransmission spectrum illustrated in FIG. 4A for each of the wavelengthband W1, the wavelength band W2, and the wavelength band Wi. Averagedtransmittance is obtained by integrating the transmission spectrum T(λ)for each wavelength band and dividing the integral value by thebandwidth of the wavelength band. In this description, a transmittancevalue averaged for each wavelength band in this manner will be referredto as transmittance in that wavelength band. In this example,transmittance is outstandingly high in the three wavelength bands havingthe maximum value P1, the maximum value P3, and the maximum value P5. Inparticular, the transmittance exceeds 0.8 in the two wavelength bandshaving the maximum value P3 and the maximum value P5.

The resolution in the wavelength direction of the transmission spectrumof each filter may be set to about the bandwidth of a desired wavelengthband. In other words, in a wavelength range including one maximum valuein a transmission spectrum curve, the width of a range having a valuegreater than or equal to an average value between a minimum valueclosest to the maximum value and the maximum value may be set to aboutthe bandwidth of the desired wavelength band. In this case, thetransmission spectrum may be decomposed into frequency components by,for example, a Fourier transform, so that the value of a frequencycomponent corresponding to the wavelength band relatively increases.

As illustrated in FIG. 2A, the filter array 10 typically has filterssegmented into a grid-like pattern. These filters partially or entirelyhave transmission spectra different from each other. Thelight-transmittance wavelength distribution and spatial distribution ofthe filters included in the filter array 10 may be, for example, arandom distribution or a semi-random distribution.

The concepts of a random distribution and a semi-random distribution areas follows. First, each filter in the filter array 10 may be regardedas, for example, a vector component having a value of 0 to 1 inaccordance with the light transmittance. In a case where thetransmittance is 0, the value of the vector component is 0. In a casewhere the transmittance is 1, the value of the vector component is 1. Inother words, a group of filters arranged in a single line in the rowdirection or the column direction may be regarded as a multidimensionalvector having a value from 0 to 1. Therefore, it may be regarded thatthe filter array 10 includes multidimensional vectors in the rowdirection or the column direction. In this case, a random distributionmeans that two freely-chosen multidimensional vectors are independent,that is, not parallel. A semi-random distribution means that themultidimensional vectors partially include a non-independentconfiguration. Therefore, in a random distribution and a semi-randomdistribution, a vector having a light transmittance value in a firstwavelength band as an element in each filter belonging to a group offilters included in the filters and arranged in one row or column and avector having a light transmittance value in the first wavelength bandas an element in each filter belonging to a group of filters arranged inanother row or column are independent from each other. With regard to asecond wavelength band different from the first wavelength band, avector having a light transmittance value in the second wavelength bandas an element in each filter belonging to a group of filters included inthe filters and arranged in one row or column and a vector having alight transmittance value in the second wavelength band as an element ineach filter belonging to a group of filters arranged in another row orcolumn are independent from each other.

In a case where the filter array 10 is disposed close to or directly onthe image sensor 60, the spacing between the filters included in thefilter array 10 may substantially match the pixel pitch of the imagesensor 60. Accordingly, the resolution of an encoded image of lightoutput from the filter array 10 substantially matches the resolution ofpixels. Light transmitted through each filter enters only a singlecorresponding pixel, so that an arithmetic process to be described latercan be readily performed. In a case where the filter array 10 isdisposed away from the image sensor 60, the pitch of the filters may beset finely in accordance with the distance.

In the examples illustrated in FIGS. 2A to 2D, the filter array 10 has agray-scale transmittance distribution in which the transmittance of eachfilter may be a freely-chosen value that is greater than or equal to 0and less than or equal to 1. However, such a gray-scale transmittancedistribution is not necessarily essential. For example, a binary-scaletransmittance distribution in which the transmittance of each filter mayhave a value of either substantially 0 or substantially 1 may beemployed. In a binary-scale transmittance distribution, each filtertransmits a large portion of light in at least two wavelength bands ofthe wavelength bands included in the target wavelength range, and doesnot transmit a large portion of light in the remaining wavelength bands.The expression “large portion” refers to substantially 80% or more.

Of all the filters, some of them, such as half of the filters, may bereplaced with transparent filters. Such transparent filters transmitlight in all the wavelength bands W1 to Wi included in the targetwavelength range with about the same high transmittance. For example,the high transmittance is higher than or equal to 0.8. In such aconfiguration, the transparent filters may be arranged in, for example,a checkboard pattern. In other words, in two arrangement directions ofthe filters in the filter array 10, filters whose light transmittancevaries in accordance with the wavelength and transparent filters may bealternately arranged. In the example illustrated in FIG. 2A, the twoarrangement directions are a horizontal direction and a verticaldirection.

Such data indicating the spatial distribution of the spectraltransmittance of the filter array 10 is preliminarily acquired based ondesign data or actual measurement calibration, and is stored in astorage medium included in the processing circuit 200. The data is usedin an arithmetic process to be described later.

The filter array 10 may be constituted by using, for example, amultilayer film, an organic material, a diffraction grating structure,or a metal-containing micro-structure. In a case where a multilayer filmis to be used, for example, a dielectric multilayer film or a multilayerfilm including a metallic layer may be used. In this case, the filterarray 10 may be formed such that at least one of the thickness, thematerial, and the stacked order of each multilayer film varies for eachfilter. Accordingly, spectral characteristics that vary from filter tofilter can be realized. By using a multilayer film, a sharp rise andfall of the spectral transmittance can be realized. A configuration thatuses an organic material may be realized by varying a contained pigmentor dye from filter to filter, or by stacking different types ofmaterials. A configuration that uses a diffraction grating structure maybe realized by providing a diffracting structure with a diffractionpitch or depth that varies from filter to filter. In a case where ametal-containing micro-structure is to be used, the filter array 10 maybe fabricated by utilizing spectroscopy based on a plasmon effect.

Processing Circuit

Next, a method for generating multi-wavelength spectral images 220 in areconstruction process by using the processing circuit 200 will bedescribed. The term “multi-wavelength” refers to, for example,wavelength bands larger in number than the three color wavelength bandsof RGB acquired by a normal color camera. The number of wavelength bandsmay be, for example, four to about 100. The number of wavelength bandsmay also be referred to as “the number of spectral bands”. Depending onthe intended usage, the number of spectral bands may exceed 100.

Data to be desirably obtained is a spectral image 220, and the data isexpressed as f. Assuming that the number of spectral bands is defined asw, f is data obtained by integrating image data f₁, f₂, . . . , andf_(w) of respective bands. As illustrated in FIG. 1A, the horizontaldirection of an image is defined as an x direction, and the verticaldirection of an image is defined as a y direction. Assuming that thenumber of pixels in the x direction of image data to be obtained isdefined as n and the number of pixels in they direction is defined as m,each piece of image data f₁, f₂, . . . , and f_(w) is two-dimensionaldata with n×m pixels. Therefore, data f is three-dimensional data withn×m×w elements. On the other hand, the number of elements in data g ofthe image 120 acquired by being encoded and multiplexed by the filterarray 10 is n×m. The data g can be expressed using Expression (1)indicated below:

$\begin{matrix}{{\mathcal{g}} = {{Hf} = {H\begin{bmatrix}f_{1} \\f_{2} \\ \vdots \\f_{w}\end{bmatrix}}}} & (1)\end{matrix}$

In this case, f₁, f₂, . . . , and f_(w) each denote data having n×melements. Therefore, a vector at the right-hand side is strictly aone-dimensional vector of n×m×w rows and one column. A vector g isexpressed and calculated by being converted into a one-dimensionalvector of n×m rows and one column. A matrix H expresses a transforminvolving encoding and intensity-modulating components f₁, f₂, . . . ,and f_(w) of a vector f with encoding information that varies for eachwavelength band and adding the components together. Therefore, H is amatrix of n×m rows and n×m×w columns.

If the vector g and the matrix H are given, it appears as if f can becalculated by solving an inverse problem of Expression (1). However,since the number of elements n×m×w in the data f to be obtained isgreater than the number of elements n×m in the acquisition data g, thisproblem is an ill-posed problem and cannot be solved as is. Theprocessing circuit 200 utilizes the redundancy of the image included inthe data f to obtain a solution by using a compressed sensing technique.In detail, the data f to be obtained is estimated by solving Expression(2) indicated below.

$\begin{matrix}{f^{\prime} = {\underset{f}{\arg\min}\left\{ {{{{\mathcal{g}} - {Hf}}}_{l_{2}} + {{\tau\Phi}(f)}} \right\}}} & (2)\end{matrix}$

In this case, f′ denotes data of estimated f. The first term in theparentheses in the above expression indicates an amount of deviation,that is, a so-called residual term, between an estimation result Hf andthe acquisition data g. Although a square sum is set as the residualterm here, an absolute value or a root-sum-square value may be set asthe residual term. The second term in the parentheses is aregularization term or a stabilization term to be described later.Expression (2) involves determining f that minimizes the sum of thefirst term and the second term. The processing circuit 200 can convergeon solution in accordance with a recursive iterative operation so as toultimately calculate a solution f′.

The first term within the parentheses in Expression (2) indicates anarithmetic process involving determining a square sum of a differencebetween the acquisition data g and Hf obtained by performing a systemconversion on fin the estimation process using the matrix H. In thesecond term, Φ(f) denotes a limiting condition in the regularization off, and is a function having sparse information of the estimation datareflected therein. With regard to the function, there is an advantage ofmaking the estimation data smooth or stable. The regularization term maybe expressed by, for example, a discrete cosine transform (DCT) of f, awavelet transform, a Fourier transform, or a total variation (TV). Forexample, if a total variation is used, stable estimation data in whichthe effect of noise in the observation data g is reduced can beacquired. The sparse characteristics of the target object 70 in thespace of each regularization term vary depending on the texture of thetarget object 70. A regularization term in which the texture of thetarget object 70 becomes sparser in the space of the regularization termmay be selected. Alternatively, regularization terms may be included inthe arithmetic process. τ denotes a weighting factor. The larger theweighting factor τ, the amount of cutback of redundant data increases,thus increasing the compression ratio. The smaller the weighting factorτ, the weaker the convergence to the solution. The weighting factor τ isset to an appropriate value at which f converges to a certain extent andthat does not lead to over-compression.

In the configurations in FIGS. 1B to 1D, an image encoded by the filterarray 10 is acquired in a blurry state on the imaging surface of theimage sensor 60. Therefore, by preliminarily storing this blurrinessinformation and reflecting this blurriness information on theaforementioned system matrix H, spectral images 220 can be generated ina reconstruction process. The blurriness information can be expressed bya point spread function (PSF). A PSF is a function that defines thedegree of spreading of a point image toward surrounding pixels. Forexample, if a point image corresponding to one pixel in an image spreadsto a k×k pixel region surrounding the pixel due to blurriness, the PSFmay be defined as a coefficient group, that is, a matrix, indicating aneffect on the pixels within the region. By reflecting the effect ofblurriness of the encoding pattern by the PSF on the system matrix H,spectral images 220 can be generated in a reconstruction process.Although the filter array 10 may be disposed at a freely-chosenposition, a position where the encoding pattern of the filter array 10does not spread too much and disappear may be selected.

Although an arithmetic example using compressed sensing indicated inExpression (2) is described here, a solution may be obtained by usinganother method. For example, another statistical method, such as amaximum likelihood estimation method or a Bayes estimation method, maybe used. Furthermore, the number of spectral images 220 is arbitrary,and the wavelength bands may also be set arbitrarily. The reconstructionmethod is disclosed in detail in U.S. Pat. No. 9,599,511. The entiredisclosure contents of U.S. Pat. No. 9,599,511 are incorporated in thisdescription.

Feature Values of Filter Array 10

An image actually captured by using the filter array 10 may containrandom noise called optical shot noise. Random noise increases inproportion to the square root of the detection light intensity. Featurevalues that the filter array 10 according to this embodiment have andthat are used for discussing random noise, to be described later, willbe described here with reference to FIGS. 5A and 5B. FIG. 5Aschematically illustrates a spatial pattern of light transmittance withrespect to each of the wavelength bands in the filter array 10. Thespatial pattern is expressed as a mosaic pattern. The number ofwavelength bands will be defined as N (N being an integer greater thanor equal to 4). As illustrated in FIG. 5A, the light-transmittancedistribution of the filter array 10 varies for each wavelength band.FIG. 5B schematically illustrates an example of a transmittancehistogram that the filter array 10 has with respect to an i-thwavelength band (i being an integer greater than or equal to 1 and lessthan or equal to N). In the example illustrated in FIG. 5B, the abscissaaxis indicates transmittance, whereas the ordinate axis denotes thenumber of filters having the transmittance. From the histogramillustrated in FIG. 5B, an average transmittance value μ_(i) of light inthe i-th wavelength band and a finite standard deviation σ_(i) areobtained as the feature values of the filter array 10. If the histogramillustrated in FIG. 5B has a Gaussian distribution, the number offilters with transmittance included within a range greater than or equalto μ_(i)−σ_(i) and less than or equal to μ_(i)+σ_(i) occupiesapproximately 68% of the total. The number of filters with transmittanceincluded within a range greater than or equal to μ_(i)−2σ_(i) and lessthan or equal to μ_(i)+2σ_(i) occupies approximately 95% of the total.As illustrated in FIG. 5B, the number of filters having hightransmittance increases as the value of μ_(i)−2σ_(i) increases. Thefilters 100 can be readily distinguished from each other as σ_(i)increases.

To obtain a histogram, the transmittance of each optical filter in thefilter array 10 is measured by using a light detector that measures thelight intensity based on a predetermined gray-scale value. For example,a histogram can be obtained by using a light detector, such as an imagesensor, capable of detecting a two-dimensional light-intensitydistribution based on a predetermined gray-scale value, such as 8 bitsor 16 bits. In detail, the transmittance of light in the i-th wavelengthband of each filter 100 in the filter array 10 can be determined fromthe ratio between the intensity of light in the i-th wavelength banddetected in a state where the filter array 10 is disposed and theintensity of light in the i-th wavelength band detected in a state wherethe filter array 10 is not disposed. The histogram as illustrated inFIG. 5B can be obtained from the data about the transmittance of eachfilter 100 acquired in accordance with the above-described method. Inthe actual filter array 10, a histogram with a shape different from thatin FIG. 5B may possibly be obtained. Because the wavelength dependencywith respect to transmittance varies from filter to filter, the shape ofthe histogram varies from wavelength band to wavelength band. Therefore,the average transmittance value of the filters 100 and the standarddeviation also vary from wavelength band to wavelength band.

In the following description, it is assumed that the gray-scale value ofthe transmittance histogram is 8 bits. Supposing that the lightintensity is detected based on a gray-scale value other than 8 bits, atransmittance histogram can still be determined by converting theintensity into the 8-bit gray-scale value.

Effect of Random Noise on Spectral Image

Next, an effect that random noise has on a spectral image will bedescribed with reference to FIGS. 6A to 7B.

FIG. 6A schematically illustrates an example of a captured image havingrandom noise superimposed on an original image. As illustrated in FIG.6A, the captured image containing the random noise may possibly beblurry, as compared with the original image. FIG. 6B schematicallyillustrates the relationship of Expression (1) in which the capturedimage data g containing the random noise is expressed as a product ofthe matrix H and the spectral image data f. The matrix H illustrated inFIG. 6B includes components of the spatial distribution of thetransmittance for each wavelength band illustrated in FIG. 5A.

FIG. 7A illustrates an example of a transmission spectrum of a certainfilter included in a filter array different from that in thisembodiment. In the example illustrated in FIG. 7A, the target wavelengthrange W is greater than or equal to 450 nm and less than or equal to 850nm. The target wavelength range W includes five wavelength bands eachhaving a bandwidth of 80 nm. In the example illustrated in FIG. 7A, thetransmission spectrum has eight sharp peaks. The transmittance has amaximum value of approximately 1 and a minimum value of approximately0.02. The transmittance between two neighboring peaks is sufficientlylow. The filters 100 having such a transmission spectrum are eachrealized by a resonant cavity including two reflective layers with highreflectance within the target wavelength range W and an interferencelayer between the reflective layers. By changing the refractive indexand/or the thickness of the interference layer, the transmissionspectrum shifts toward the longer wavelength side or the shorterwavelength side. The filter array includes various types of filters withtransmission spectra that have been shifted in this manner. The filterarray includes 1,000,000 two-dimensionally-arranged filters. The1,000,000 filters include 16 types of filters arranged in the randomdistribution or the semi-random distribution.

FIG. 7B illustrates a mean squared error (MSE) between a ground truthimage and a spectral image in each of five wavelength bands in a casewhere an image captured using the aforementioned filter array containsrandom noise. The random noise has occurred in accordance with aGaussian distribution in which the center is μ_(noise)=0 and thestandard deviation is σ_(noise). A random-noise variance N_(noise) to beadded to each pixel is calculated by using Expression (3) indicatedbelow:

$\begin{matrix}{N_{noise} = {\frac{1}{\sqrt{2{\pi\sigma}_{noise}}}{\exp\left( {- \frac{\left( {{\mathcal{g}}_{noise} - \mu_{noise}} \right)^{2}}{2{\sigma_{noise}}^{2}}} \right)}}} & (3)\end{matrix}$

In this case, g_(noise) denotes a value of noise superimposed on eachpixel and may be a positive value or a negative value. In the exampleillustrated in FIG. 7B, σ_(noise)=0, 5, and 10. σ_(noise)=0 implies thatthere is no random noise. The random noise increases with increasingσ_(noise). The mean squared error is calculated by using Expression (4)indicated below.

$\begin{matrix}{{MSE} = {\frac{1}{N \cdot M}{\sum\limits_{i = 1}^{N}{\sum\limits_{j = 1}^{M}\left( {I_{i,j}^{\prime} - I_{i,j}} \right)^{2}}}}} & (4)\end{matrix}$

In this case, N and M denote the number of pixels in the horizontaldirection and the vertical direction, respectively. I_(i,j) denotes apixel value of a pixel at a position (i, j) in the ground truth image.I′_(i,j) denotes a pixel value of a pixel at the position (i, j) in thespectral image.

As illustrated in FIG. 7B, it is apparent that the MSE in all of thefive wavelength bands increases with increasing σ_(noise). If there isno random noise, a filter having a transmission spectrum with a largecontrast ratio between the maximum value and the minimum value of thetransmittance can achieve an improved S/N ratio between the wavelengthbands, as illustrated in FIG. 7A. In other words, the difference in thespatial distribution of the transmittance between the wavelength bands,illustrated in FIG. 5A, becomes clear. However, a filter array includingsuch filters is susceptible to random noise. As a result, the spectralresolution of a hyperspectral camera may deteriorate.

Filter Array 10 Having Noise Immunity

The following description relates to the filter array 10 according tothis embodiment that can solve the aforementioned problem. The filterarray 10 in which the value of μ_(i)−2σ_(i) illustrated in FIG. 5B islarge has noise immunity. This is because the transmittance is high insuch a filter array 10. The filter array 10 in which σ_(i) illustratedin FIG. 5B is large also has noise immunity. This is due to improvedspectral-image reconstruction accuracy since the variance oftransmittance between the filters is large and the filters 100 can thusbe readily distinguished from each other in such a filter array 10.Accordingly, the optical characteristics of the filter array 10according to this embodiment will now be described by usingM_(i)=μi−σ_(i), which is a sum of μ_(i)−2σ_(i) and σ_(i), as anindicator of noise immunity. Similar to the above-described example, thefilter array 10 according to this embodiment includes various types offilters arranged in the random distribution or the semi-randomdistribution.

FIG. 8 illustrates the relationship between an average MSE between aground truth image and a spectral image in each of the wavelength bandsand M_(i)=μi−σ_(i) in a case where σ_(i)=0.08 with respect to eachwavelength band. In the example illustrated in FIG. 8 , the random-noisestandard deviation σ_(noise)=0, 2, 3, 4, and 5. The amount of lighttransmitted through the filter array 10 is dependent on M_(i). Theamount of light transmitted through the filter array 10 decreases as thevalue of M_(i) decreases, whereas the amount of light transmittedthrough the filter array 10 increases as the value of M_(i) increases.If the filters 100 included in the filter array 10 mostly have lowtransmittance and partially have high transmittance, the transmittancehistogram of the filter array 10 may possibly be asymmetric. In such afilter array 10, M_(i) may possibly be a negative value.

In a case where σ_(noise)=0, a captured image does not contain randomnoise. In this case, the average MSE is kept low when M_(i)≤0.45. Asillustrated in FIG. 7A, if the contrast ratio of the transmittance inthe filters 100 is high, an average amount of light transmitted throughthe filter array 10 is small, and the value of M_(i) is small. Such afilter array 10 increases the S/N ratio between the wavelength bands. Asa result, an occurrence of a spectral-image reconstruction error can besuppressed. In contrast, when M_(i)≥0.45, the average MSE increases asM_(i) increases. If the contrast ratio of the transmittance in thefilters 100 is low, the average amount of light transmitted through thefilter array 10 is large, and the value of M_(i) is large. Such a filterarray 10 decreases the S/N ratio between the wavelength bands. As aresult, the spectral-image reconstruction error increases.

In a case where σ_(noise)>0, the average MSE increases greatly asσ_(noise) increases when M_(i)<0.2, as indicated with an ellipticalregion surrounded by a solid line. In contrast, when M_(i) is near 0.42,the average MSE increases as σ_(noise) increases, but the amount ofincrease thereof is suppressed, as indicated with an elliptical regionsurrounded by a dashed line. When M_(i)≥0.45, the average MSE increasesas M_(i) increases even if an is small. As illustrated in FIG. 8 , in acase where σ_(noise)=5, the average MSE may sometimes increase suddenlyin a discontinuous fashion. The above characteristics are summarized asfollows.

-   -   (If M_(i) is small) the S/N ratio between the wavelength bands        is high. Therefore, when σ_(noise) is small, the spectral-image        reconstruction accuracy is high. When σ_(noise) increases, the        amount of random noise is not negligible, as compared with a        small amount of light transmitted through the filter array 10.        As a result, the spectral-image reconstruction accuracy greatly        decreases.    -   (If M_(i) is too large) the S/N ratio between the wavelength        bands is low. Therefore, the spectral-image reconstruction        accuracy is low even if σ_(noise) is small.    -   (If M_(i) is reasonably large) the spectral-image reconstruction        accuracy is high when σ_(noise) is small. Even if σ_(noise)        increases, the amount of random noise does not increase much, as        compared with the amount of light transmitted through the filter        array 10. As a result, a decrease in the spectral-image        reconstruction accuracy is suppressed.

FIGS. 9A to 9C each illustrate the relationship between the average MSEbetween a ground truth image and a spectral image in each of thewavelength bands and M_(i)=μ_(i)−σ_(i) in a case where σ_(i)=0.05, 0.07,and 0.1 with respect to each wavelength band. As illustrated in FIGS. 9Ato 9C, in a case where M_(i) is reasonably large, a decrease in thespectral-image reconstruction accuracy is suppressed even when anincreases. Such M_(i) is greater than or equal to approximately 0.3 andless than or equal to approximately 0.5. In actuality, there is areconstruction error caused by other noise, such as read noise, of aboutseveral %, in addition to the reconstruction error caused by randomnoise. When the total reconstruction error exceeds 5%, the lowness ofthe spectral-image reconstruction accuracy may possibly become visible.Therefore, a permissible reconstruction error caused by random noise maypossibly be, for example, more than or equal to 3%. In an 8-bitgray-scale, a reconstruction error of 3% is equal to an average MSE of56. In the examples illustrated in FIGS. 9A to 9C, in a case whereσ_(noise)=5 that resembles actual random noise, the lower limit forM_(i) when the average MSE falls below 56 is M_(i)=0.1. Accordingly, itmay be regarded that the filter array 10 having M_(i)≥0.1 is robustagainst random noise.

In addition to having M_(i)≥0.1, the filter array 10 according to thisembodiment has the following characteristics.

-   -   σ_(i)≥0.05 with respect to each of the wavelength bands. In this        case, since the filters 100 are readily distinguishable from        each other, the spectral-image reconstruction accuracy can be        improved.    -   μ_(i)≥0.2 with respect to each of the wavelength bands. In this        case, M_(i)≥0.1 is satisfied with respect to 0.05≤σ_(i)≤0.1        illustrated in FIGS. 8 to 9C.    -   M_(max)/M_(min)≤3, where a maximum M_(i) and a minimum M_(i)        with respect to the wavelength bands are defined as M_(max) and        M_(min), respectively. In this case, a variation in the        spectral-image reconstruction accuracy between the wavelength        bands can be suppressed.

Specific Structure of Filter Array

Next, an example of a specific structure of the filter array 10according to this embodiment that satisfies M_(i)≥0.1 with respect toeach of the wavelength bands will be described with reference to FIGS.10A and 10B. FIGS. 10A and 10B are cross-sectional views schematicallyillustrating a first example and a second example, respectively, of thelight detection device 300. Each cross-sectional view illustrates anexample of a cross-sectional structure of one of the rows of the filterarray 10 illustrated in FIG. 2A and the image sensor 60. In the examplesillustrated in FIGS. 10A and 10B, the filter array 10 is disposed on theimage sensor 60. Light detection elements 60 a included in the imagesensor 60 are respectively positioned directly below the filters 100included in the filter array 10. The filter array 10 and the imagesensor 60 may be separated from each other. Even in that case, eachlight detection element 60 a may be disposed at a position where itreceives light transmitted through one of the filters. The componentsmay be disposed such that light transmitted through the filters entersthe light detection elements 60 a via a mirror. In that case, each lightdetection element 60 a is not disposed directly below one of thefilters.

The filters 100 included in the filter array 10 according to thisembodiment each have a structure of a resonant cavity. The structure ofthe resonant cavity refers to a structure in which light with a certainwavelength forms a standing wave and exists stably therein. A resonantcavity illustrated in FIG. 10A includes a substrate 22, a reflectivelayer 24, and an interference layer 26 stacked in this order. A resonantcavity illustrated in FIG. 10B includes a substrate 22, an interferencelayer 26, and a reflective layer 24 stacked in this order. The substrate22 illustrated in each of FIGS. 10A and 10B is provided uniformlythroughout all the filters 100 without any steps. Unlike the reflectivelayer 24 illustrated in FIG. 10B, the reflective layer 24 illustrated inFIG. 10A is provided uniformly throughout all the filters 100 withoutany steps. As illustrated in FIGS. 10A and 10B, the interference layer26 and the reflective layer 24 may be stacked in a freely-chosen order.The substrate 22 is not necessarily essential. The reflective layer 24may include, for example, a distributed Bragg reflector (DBR). Theconfiguration of the reflective layer 24 will be described in detaillater. The interference layer 26 has a different refractive index and/ora different thickness for each filter 100. The filters 100 havedifferent transmission spectra depending on the refractive index and/orthe thickness of the interference layer 26. The transmission spectrum ofeach filter 100 has maximum transmittance values at various wavelengthswithin the target wavelength range W.

In the filter array 10 according to this embodiment, at least one filterof the filters 100 may have the aforementioned resonant cavity, whereasthe other filters do not have to have the aforementioned resonantcavity. For example, the filter array 10 may include a filter, such as atransparent filter or an ND filter (neutral density filter), not havingwavelength dependency with respect to light transmittance.

In the first example, the interference layer 26 may be exposed toatmospheric air. Components, such as a lens and a protection cover, maybe disposed above the surface of the interference layer 26 with a spacetherebetween. In this case, the space may be filled with atmosphericair, or may be sealed with gas, such as nitrogen gas. The same appliesto the surface of the reflective layer 24 in the second example.

Next, an example of the configuration of each filter 100 will bedescribed with reference to FIG. 11A. FIG. 11A schematically illustratesan example of each filter 100 illustrated in FIG. 10A. As illustrated inFIG. 11A, the reflective layer 24 includes a DBR in which firstrefractive-index layers 24 a and second refractive-index layers 24 b arealternately stacked. The DBR includes one or more pair layers of firstrefractive-index layers 24 a and second refractive-index layers 24 bhaving different refractive indices. The refractive index of the firstrefractive-index layers 24 a is higher than the refractive index of thesecond refractive-index layers 24 b. A distributed Bragg reflector has awavelength range with high reflectance in accordance with Braggreflection occurring due to a periodical structure. Such a wavelengthrange is also called a stop band. When the number of aforementioned pairlayers is increased, the reflectance in the stop band approaches 100%.

A wavelength within the target wavelength range W will be defined as λ,the refractive index of the first refractive-index layers 24 a will bedefined as n_(H), and the refractive index of the secondrefractive-index layers 24 b will be defined as n_(L). A DBR includingone or more pair layers of first refractive-index layers 24 a having athickness of λ/(4n_(H)) and second refractive-index layers 24 b having athickness of λ/(4 n_(L)) efficiently reflects light with the wavelengthλ. If the target wavelength range W is a range greater than or equal toa wavelength λ_(i) and less than or equal to a wavelength λ_(f), the DBRcan include a pair layer corresponding to the wavelength λ_(i) to a pairlayer corresponding to the wavelength by varying the thicknesses of thefirst refractive-index layers 24 a and the second refractive-indexlayers 24 b in a stepwise fashion. As a result, the DBR can efficientlyreflect all the light within the target wavelength range W.

The DBR may be composed of, for example, a material with low absorbancewith respect to light within the target wavelength range W. If thetarget wavelength range W is within the visible-light region, thematerial may be at least one selected from the group consisting of SiO₂,Al₂O₃, SiO_(x)N_(y), Si₃N₄, Ta₂O₅, and TiO₂. If the target wavelengthrange W is within the infrared region, the material may be at least oneselected from the group consisting of monocrystalline Si,polycrystalline Si, and amorphous Si, in addition to SiO₂, Al₂O₃,SiO_(x)N_(y), Si₃N₄, Ta₂O₅, and TiO₂ mentioned above.

The interference layer 26 has a lower surface 26 s ₁ in contact with thereflective layer 24 and also has an upper surface 26 s ₂ at the oppositeside thereof. In the filter 100 illustrated in FIG. 11A, the lowersurface 26 s ₁ corresponds to a first surface, and the upper surface 26s ₂ corresponds to a second surface. In the example illustrated in FIG.11A, the upper surface 26 s ₂ is exposed to the outside and is incontact with the air. A transparent layer may be further stacked on theinterference layer 26. In this case, the upper surface 26 s ₂ is incontact with the transparent layer. The reflectance (referred to as“first reflectance” hereinafter) of the lower surface 26 s ₁ withrespect to light in the target wavelength range W may be, for example,higher than or equal to 80%. The first reflectance may be lower than80%, but may be designed to be higher than or equal to 40% from thestandpoint of suppressing reflection. The reflectance (referred to as“second reflectance” hereinafter) of the upper surface 26 s ₂ withrespect to light in the target wavelength range W is lower than thefirst reflectance and may be, for example, higher than or equal to 1%and lower than 30%. There is a fixed difference of 10% or more betweenthe first reflectance and the second reflectance.

On the other hand, in the filters 100 of the light detection device 300illustrated in FIG. 10B, the reflective layer 24 is disposed over theupper surface of the interference layer 26. In the filters 100 of thelight detection device 300 illustrated in FIG. 10B, the upper surface ofthe interference layer 26 corresponds to the first surface, and thelower surface corresponds to the second surface.

In this description, unless an accurate position of a surface thatreflects light becomes a problem, the light within the interferencelayer 26 is reflected at the lower surface 26 s ₁ and the upper surface26 s ₂. In this embodiment, a portion of light incident on thereflective layer 24 from the interference layer 26 actually enters thereflective layer 24 so as to be reflected at the interfaces between thefirst refractive-index layers 24 a and the second refractive-indexlayers 24 b. The interfaces where the light is reflected vary dependingon the wavelength. However, for the sake of convenience, these beams oflight are treated as being reflected at the lower surface 26 s ₁.

The reflection of the light at the lower surface 26 s ₁ and the uppersurface 26 s ₂ causes standing waves to be formed within theinterference layer 26. As a result, if the thickness of the interferencelayer 26 is greater than or equal to a predetermined value, thetransmission spectrum of each filter 100 has maximum transmittancevalues at various wavelengths in the target wavelength range W. In otherwords, the transmission spectrum of the filter 100 has peaks within thetarget wavelength range W. In this description, such a filter will bereferred to as “multimode filter”. If the DBR includes a pair layercorresponding to the wavelength λ, the thickness of the interferencelayer 26 that can realize a multimode filter may be, for example, twicethe thickness of the first refractive-index layers 24 a, that is,greater than or equal to λ/(2n_(H)). A multimode filter may be realizedby appropriately designing the refractive index of the interferencelayer 26 instead of the thickness of the interference layer 26.Alternatively, a multimode filter may be realized by appropriatelydesigning both the refractive index and the thickness of theinterference layer 26.

The interference layer 26 may be composed of a material similar to thatof the DBR. The interference layer 26 is not limited to a single layerand may include stacked layers. Such layers may be composed of differentmaterials. The layers may have different refractive indices to an extentthat they do not have a substantial effect on the transmission spectrumof each filter 100. Reflection may occur at the interfaces betweenlayers with different refractive indices. However, the layers may beregarded as a substantially uniform part of the interference layer 26 solong as they do not have a substantial effect on the transmissionspectrum. A permissible relative error in the refractive indices isgreater than or equal to 0% and less than or equal to 9%. The relativeerror is a value obtained by dividing an absolute value of a differencebetween a maximum refractive index and a minimum refractive index by themaximum refractive index. For example, the refractive indices in thevisible-light regions of Ta₂O₅ and Si₃N₄ are 2.2 and 2.05, respectively.The relative error in these refractive indices is about 7%. Therefore,the stacked Ta₂O₅ and Si₃Na layers may be regarded as a substantiallyuniform part of the interference layer 26.

In the following description, the structure illustrated in FIG. 11A willbe referred to as “single-sided DBR structure”. In addition to thestructure illustrated in FIG. 11A, a structure having a reflective layer24 additionally stacked on the interference layer 26 will be referred toas “double-sided DBR structure”. In the single-sided DBR structure, thereflective layer 24 is provided on one of the lower surface 26 s ₁ andthe upper surface 26 s ₂ of the interference layer 26, but is notprovided on the other surface. In the double-sided DBR structure,reflective layers are provided on both the lower surface 26 s ₁ and theupper surface 26 s ₂ of the interference layer 26.

Next, an example of transmission spectra of the filters 100 will bedescribed with reference to FIG. 11B. FIG. 11B illustrates an example oflight transmission spectra of the single-sided DBR structure and thedouble-sided DBR structure when light vertically enters the structures.In the example illustrated in FIG. 11B, the target wavelength range W isgreater than or equal to 450 nm and less than or equal to 850 nm. Asolid line illustrated in FIG. 11B indicates the transmission spectrumof the single-sided DBR structure. A dashed line in FIG. 11B indicatesthe transmission spectrum of the double-sided DBR structure. Asillustrated in FIG. 11B, multimode filters can be obtained from both thesingle-sided DBR structure and the double-sided DBR structure. However,the characteristics of the multimode filters of the two structures aredifferent from each other with respect to the following points. Thetransmission spectrum of the single-sided DBR structure has wide peaksand has maximum transmittance of about 0.5 and minimum transmittance ofabout 0.1. In other words, the contrast ratio of the transmittance islow. In contrast, the transmission spectrum of the double-sided DBRstructure has sharp peaks and has maximum transmittance of about 1.0 andminimum transmittance of about 0.02. In other words, the contrast ratioof the transmittance is high.

In the single-sided DBR structure, the base line of the transmittancerises, as compared with the double-sided DBR structure. In this example,the average transmittance within the target wavelength range W in thesingle-sided DBR structure is approximately 0.26, and the averagetransmittance within the target wavelength range W in the double-sidedDBR structure is approximately 0.1. In the single-sided DBR structure,the average transmittance is about twice that in the double-sided DBRstructure. Accordingly, the single-sided DBR structure can suppress aloss in light intensity during an imaging process.

In the example illustrated in FIG. 11B, in the single-sided DBRstructure and the double-sided DBR structure, M_(i)=0.1 and M_(i)=−0.14,respectively, with respect to each of the wavelength bands. If each ofthe filters 100 included in the filter array 10 includes thesingle-sided DBR structure, M_(i)≥0.1 can be satisfied with respect toeach wavelength band.

Next, an example of a specific structure of the filter array 10according to this embodiment that satisfies M_(i)≥0.1 with respect toeach of the wavelength bands will be described with reference to FIGS.12A and 12B. FIGS. 12A and 12B are cross-sectional views schematicallyillustrating a third example and a fourth example, respectively, of thelight detection device 300. The third example and the fourth example aredifferent from the first example and the second example in that areflective layer 25 includes a metallic film instead of a DBR. Thereflectance of the metallic film in the target wavelength range W isdependent on the material of the metallic film. An absorptioncoefficient of the metallic film in the target wavelength range W isdependent on the thickness of the metallic film.

The metallic film may be composed of a material whose reflectance in thetarget wavelength range W is higher than or equal to 90%. If the targetwavelength range W is within the visible-light region, the material ofthe metallic film with the reflectance being higher than or equal to 90%may be at least one selected from the group consisting of Ag and Al. Ifthe target wavelength range W is within the infrared region, thematerial of the metallic film with the reflectance being higher than orequal to 90% may be at least one selected from the group consisting ofAg, Al, Au, and Cu. Even if the metallic film has relatively lowreflectance, the metallic film is still useful in that the reflectanceis suppressed. For example, the metallic film may be composed of amaterial whose reflectance in the target wavelength range W is higherthan or equal to 40% and lower than or equal to 70%. If the targetwavelength range W is within the visible-light region or the infraredregion, the material of the metallic film with the reflectance beinghigher than or equal to 40% and lower than or equal to 70% may be atleast one selected from the group consisting of Ni and Pt. The metallicfilm may be composed of an alloy. The metallic film may be provided byplating.

The thickness of the metallic film may be, for example, larger than orequal to 1 nm and smaller than or equal to 100 nm. In this case, thetransmittance of the metallic film from the visible-light region to theinfrared region may relatively increase. As a result, light in thevisible light region to the near-infrared region can be effectivelytransmitted through the metallic film. The thickness of the metallicfilm may be smaller than or equal to several tens of nm.

In the examples illustrated in FIGS. 10A and 10B and FIGS. 12A and 12B,the reflective layer exists only at one side of the interference layer26. If the first reflectance and the second reflectance of theinterference layer 26 have a fixed difference, as described above,reflective layers may exist at both sides of the interference layer 26.

Next, another example of the specific structure of the filter array 10according to this embodiment that satisfies M_(i)≥0.1 with respect toeach of the wavelength bands will be described with reference to FIGS.13A and 13B.

FIG. 13A is a perspective view schematically illustrating a fifthexample of the light detection device 300. In the example illustrated inFIG. 13A, the filter array 10 includes filters 110. Each of the filters110 is a sub filter array in which sub filters 30 are arranged. Thefilter array 10 is disposed such that light transmitted through thefilters 110 enters each of the light detection elements 60 a in theimage sensor 60. In the example illustrated in FIG. 13A, one lightdetection element 60 a detects light transmitted through 4×4 sub filters30. Not all of the filters have to be sub filter arrays. The filterarray according to this embodiment may include at least one sub filterarray.

FIG. 13B is a perspective view schematically illustrating an example ofeach filter 110 serving as a sub filter array and the correspondinglight detection element 60 a. The sub filters 30 include various typesof first sub filters 30 a with different transmission wavelength ranges,and also include at least one second sub filter 30 b that transmitslight in each wavelength band. In the example illustrated in FIG. 13B, ahatched sub filter indicates a first sub filter 30 a, and a white subfilter indicates a second sub filter 30 b. The number of first subfilters 30 a and the number of second sub filters 30 b vary for eachfilter 110 serving as a sub filter array. The filters 110 serving as subfilter arrays have different transmission spectra.

The various types of first sub filters 30 a may include, for example,RGB color filters, namely, a red filter, a green filter, and a bluefilter. In place of the RGB color filters or in addition to the RGBfilters, the various types of first sub filters 30 a may include atleast one of CMY filters, namely, a cyan filter, a magenta filter, and ayellow filter. The second sub filter 30 b may be, for example, atransparent filter having high transmittance in the visible-lightregion. In place of the second sub filter 30 b, an opening may beprovided. The baseline for the transmittance of the filter 110illustrated in FIG. 13B can be increased by the second sub filter 30 bor the opening. As a result, the filter array 10 illustrated in FIG. 13Acan satisfy M_(i)>0.1 with respect to each of the wavelength bands. Sucha filter array 10 has noise immunity. The percentage of an area of thesecond sub filter 30 b or the opening occupying the total area of eachfilter 110 serving as a sub filter array is higher than or equal to 4%.The percentage of 4% corresponds to a case where, for example, 5×5 subfilters 30 include one second sub filter 30 b.

Next, an example of transmission spectra of the filters 110 serving assub filter arrays will be described with reference to FIG. 14 . FIG. 14illustrates the transmission spectra of eight types of sub filterarrays. The 4×4 sub filters 30 in each filter 110 serving as a subfilter array includes 14 first sub filters 30 a having three types ofRGB color filters arranged therein and two second sub filters 30 b thattransmit white light. The percentage of an area of the second subfilters 30 b occupying the total area of the filter 110 serving as a subfilter array is 12.5%. With the second sub filters 30 b, the minimumtransmittance in the transmission spectra of all the sub filter arraysis higher than or equal to 0.15. As a result, M_(i) is equal to about0.18 with respect to each of the wavelength bands, so that M_(i)>0.1 issatisfied. In contrast, if all the 4×4 sub filters 30 in each sub filterarray are first sub filters 30 a, M_(i) is equal to about 0.05 withrespect to each of the wavelength bands, so that M_(i)>0.1 is notsatisfied.

If M_(i) is too large, the sub filters 30 may partially include an NDfilter, so that the minimum transmittance value of each filter 110serving as a sub filter array can be reduced. As a result, the value ofM_(i) decreases.

INDUSTRIAL APPLICABILITY

The light detection system and the filter array according to the presentdisclosure may be used in, for example, a camera and a measurementdevice that acquire a multi-wavelength two-dimensional image. The lightdetection system and the filter array according to the presentdisclosure are also applicable to, for example, biological, medical, orcosmetic-oriented sensing, a system for inspecting foreign matter andresidual pesticides in food, a remote sensing system, and a vehicularsensing system.

What is claimed is:
 1. A filter array used in a light detection devicethat generates image data corresponding to each of N wavelength bandsincluded in a specific wavelength range, N being an integer greater thanor equal to 2, the filter array comprising: optical filters, wherein theoptical filters include optical filters with different lighttransmittance with respect to each of the N wavelength bands, andwherein M_(i)≥0.1 with respect to each of the N wavelength bands, whereM_(i)=μ_(i)−σ_(i), μ_(i) denoting an average value of lighttransmittance of the optical filters with respect to light having awavelength included in an i-th wavelength band of the N wavelengthbands, i being an integer greater than or equal to 1 and less than orequal to N, σ_(i) denoting a standard deviation of the lighttransmittance of the optical filters with respect to the light havingthe wavelength included in the i-th wavelength band.
 2. The filter arrayaccording to claim 1, wherein σ_(i)≥0.05 with respect to each of the Nwavelength bands.
 3. The filter array according to claim 1, whereinμ_(i)≥0.2 with respect to each of the N wavelength bands.
 4. The filterarray according to claim 1, wherein M_(max)/M_(min)≤3, where M_(max) andM_(min) denote a maximum M_(i) and a minimum M_(i), respectively, withrespect to the N wavelength bands.
 5. The filter array according toclaim 1, wherein at least one optical filter of the optical filtersincludes an interference layer having a first surface and a secondsurface opposite the first surface, and a reflective layer provided onthe first surface, wherein a transmission spectrum of the at least oneoptical filter has maximum values in the specific wavelength range, andwherein the reflective layer is not provided on the second surface. 6.The filter array according to claim 5, wherein the reflective layerincludes at least one selected from the group consisting of adistributed Bragg reflector and a metallic film.
 7. The filter arrayaccording to claim 6, wherein the distributed Bragg reflector includesat least one set of a first refractive-index layer and a secondrefractive-index layer, and wherein a refractive index of the firstrefractive-index layer is higher than a refractive index of the secondrefractive-index layer.
 8. The filter array according to claim 7,wherein the first refractive-index layer has a thickness of λ/(4n_(H)),and the second refractive-index layer has a thickness of λ(4 n_(L)), andwherein the interference layer has a thickness larger than λ/(2n_(H)),where λ denotes a wavelength included in the specific wavelength range,n_(H) denotes the refractive index of the first refractive-index layer,and n_(L) denotes the refractive index of the second refractive-indexlayer.
 9. The filter array according to claim 6, wherein the metallicfilm has a thickness larger than or equal to 1 nm and smaller than orequal to 100 nm.
 10. The filter array according to claim 1, wherein thelight detection device includes an image sensor, wherein the filterarray is disposed such that light transmitted through the opticalfilters enters the image sensor, wherein at least one of the opticalfilters is a sub filter array including two-dimensionally-arranged subfilters, wherein the sub filter array includes first sub filters havingdifferent transmission wavelength ranges and one or more second subfilters or openings each transmitting N light beams having a wavelengthincluded in a corresponding wavelength band of the N wavelength bands,and wherein a percentage of an area of the one or more second subfilters or openings occupying a total area of the sub filter array ishigher than or equal to 4%.
 11. The filter array according to claim 10,wherein the first sub filters include a red filter, a green filter, anda blue filter.
 12. A light detection system comprising: the filter arrayaccording to claim 1; and an image sensor that is disposed at a positionwhere the image sensor receives light transmitted through the opticalfilters and that has sensitivity to light having a wavelength includedin the specific wavelength range.
 13. The light detection systemaccording to claim 12, further comprising: a processing circuit thatgenerates the image data based on data indicating a spatial distributionof the light transmittance of the optical filters and compressed imagedata acquired by the image sensor.